Skip to content

Bound pathological addFile() cost by truncating REFS tokenization - #4994

Open
VKspyder2003 wants to merge 1 commit into
oracle:masterfrom
VKspyder2003:4550-limit-addfile-processing-time
Open

Bound pathological addFile() cost by truncating REFS tokenization#4994
VKspyder2003 wants to merge 1 commit into
oracle:masterfrom
VKspyder2003:4550-limit-addfile-processing-time

Conversation

@VKspyder2003

Copy link
Copy Markdown

Summary

Limit pathological REFS tokenization by introducing a maximum emitted token count in JFlexTokenizer.

The limit is applied only to the REFS field used for symbol indexing. Other indexing behavior, including FULL, DEFS, xref generation, and metadata indexing, remains unchanged.

Changes

  • Add optional emitted-token limit to JFlexTokenizer
  • Apply the limit only when indexing the REFS field
  • Preserve existing indexing flow and document creation
  • Add tokenizer tests
  • Add a regression test using the large JavaScript fixture

Fixes #4550

@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Required At least one contributor does not have an approved Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

limit addFile() processing time

1 participant